Video is the technology of electronically capturing, recording, processing, storing, transmitting, and reconstructing a sequence of still images representing scenes in motion. Video technology was first developed for television systems, but has been further developed in many formats to allow for viewer video recording. Motion pictures on film can be converted into video formats. Video can also be viewed through the Internet (World Wide Web) as downloaded video files or streaming files on computer monitors.
Animation is the rapid display of a sequence of images of artwork or model positions in order to create an illusion of movement. It is an optical illusion of motion due to the phenomenon of persistence of vision, and can be created and demonstrated in a number of ways. The most common method of presenting animation is as a motion picture or video, although several other forms of presenting animation also exist.
Video content segmentation is the systematic decomposition of a motion picture frame into its objects (components) such as a person, a shirt, a tree, a leave etc. Segmenting video content results in a large number of objects with little value if not classified.
Classification is the process of assigning an object of one frame to the same class of the same object of another frame. It enables the automated recognition that a specific red shirt in one frame is the same as the red shirt in another frame. There are several approaches to assigning video objects to the class they belong to such as by the contours of its appearances in successive video frames. For example, this may be done by matching curvature features of the video object contour to a database containing preprocessed views of prototypical objects. See, Attachment 1 entitled MOCA Project Object Recognition.
For each two-dimensional appearance of an object in a video frame curvature features of its contour are calculated. These features are matched to those of views of prototypical video objects stored in a database. By applying context rules such as “a house may have a car in the frame or may have a tree in the frame but does not have a TV in the frame” the accuracy can be increased. The final classification of the object is achieved by integrating the matching results for successive frames.
There are several paradigms and algorithms for video segmentation and classification. Most are based on segmenting video into layers such as a static background layer and a dynamic foreground layer and using multiple cues, such as spatial location, color, motion, contours and depth discontinuities, etc.
Rotoscoping is an animation technique in which animators trace over live action film movement, frame by frame, for use in animated films. Digital Rotoscoping as taught by Tostevin et. al. in U.S. Pat. No. 6,393,134 uses algorithms to create vector outlines of video objects.
By shooting video from several perspectives with synchronized cameras, video segmentation algorithms can be used to automatically reconstruct 3D wireframes of moving objects. If video segments show multiple perspective of one video object, 3D wire frames can be constructed even if only one camera was used as long as the camera captured many perspectives of the object.
In one embodiment of the invention automatic rotoscoping techniques are applied to videos which have been shot by multiple camera angles to reconstruct the 3D objects and save their wireframes in form of vector data into the video object database. In another embodiment rotoscoping techniques are applied to videos which have been shot by one camera but many perspectives of the video objects are available. When a viewer selects an object for which there is 3D information available, the viewer is presented with a means to control the animation of the 3D object such as rotate, move, scale etc. In yet another embodiment of the invention animated 3D objects are positioned into videos to replace or superimpose video objects. These animated 3D video objects may have been derived from digital rotoscoping or may be animated objects from scratch.
An object of the invention is to provide an automated system for segmenting raw video to create an inventory of video objects which may be used to make the video interactive, and to auction these video objects to advertisers. The term video object is defined as data and visualizations of data which relate to content objects of videos such as a shirt. This data may include image data, or vector graphic data which has been linked to the content object of a video or has been generated from video data.
The invention is not tied to any specific method of segmenting or classifying video content objects.
In one embodiment of the invention an object information library which contains descriptive information and/or meta-data regarding objects which may appear in the video is used to associate meta data such as product information, unique product identification information or stock keeping units with the segmented video objects.
A further object of the invention is to create an advertising market exchange whereby rights to an inventory of video objects are automatically auctioned to a third party such as an advertiser.